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SUMMARY 



In this paper we discuss issues related to tvco concerns of the 
Study Group on National Assessment: (1) What should be the roles and 
responsibilities of federal and state education agencies in 
reformulating NAEP to facilitate comparisons of states? and (2) What 
are some problems and solutions for aligning or accounting for 
differences between national assessment objectives and state and local 
curricula? in the course of considering these concerns we postulate 
that NAEP will come to be viewed as a "national achievement test" and 
that NAEP objectives will be perceived as a "national curriculum," Vte 
believe that these perceptions will result from using NAEP assessments 
to make state-by-state comparisons, in addition, we discuss the 
importance of '^curriculuni coherence" to effective instruction and 
learning and in iralation to the use of NAEP for comparing states. 

Our primary recommendations to the study group, bidders on the 
new NAEP contract, state and local education agencies, and the 
Department of Education include: 

1, NAEP should be reformulated not only to (a) facilitate state 
comparisons, but also to (b) preserve and enhance curriculum 
coherence and (c) encourage improveir»ent in instruction and 
learning, 

2, Revisions of NAEP objectives, which are used to generate NAEP 
exercises, and integration of NAEP objectives into state and 
local curricula should be done in ways that preserve and 
enhance curriculum coherence. 



3. Consensus on which NAEP objectives should be assessed and used 
for stat€i-by-state comparisons should be reached only with 
input from representatives of state and local education 
agencies, 

4. Future NAEP reports of NAEP results (e.g., The Reading Report 
Card ) should include descriptions of the liHks between naep 
results and objectives assessed by NAEP. 

Finally, in the course of describing likely responses by state 
and local educcition agencies to being compared to other states, we 
suggest further issues for consideration. 
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On occasion, U.S. Department of Education (ED) Secretary William 
?: ^^-yi^^'s "Wall Chart" (U.S. DOE, 1985) has been described as a 
bad Idea done badly." The Wall Chart, which was last released in 
January_1985, ranks all states in three areas: resource inputs like 
per pupil expenditure, student performance outcomes sucTT^^ 
Scholastic Aptitude Test (SAT) and American College Ttesting (ACT) 
Program scores, and student population characteristics such perceiAt 
of minority enrollment:. Many critics consider it unfair to compare 
states on variables like SAT and ACT scores, since student 
demographics, s-ich as the percentage of high school seniors in a statp 
who actually take the SAT (Powell & Steelman, 1984) , and local 
curricula vary widely from state to state and so affect performance on 
these tests. 

■nie Council of Chief State School Officers (CCSSO) apparently 
feels that comparisons among states can be done more effectively 
Duringits 1984 Annual Meeting, the chiefs established their state 
Education Assessment Center and began discussions to standardize data 
collection procedures and choose the variables — including 
achievement indicators — on which states could be compared, r^ich of 
the Center's efforts have focused on the possibility of using 
objectives and test exercises from National Assessment of Educational 
Progress (NAEP) assessments for these comparisons. Even Secretary 
Bennett has indicated his desire to improve the Wall Chart. He has 
criticized NAEP as it is currently constituted and charged present and 
future holders of the NAEP contract to reformulate NAEP to be more 
suitable for making state-by-state comparisons (Hertling, 1986) 
Presunably, once this takes place he will be able to incorporate NAEP 
results into the Wall Chart. 

xto facilitate this effort, a 22-inember national task force was 
established to study problems and solutions related to national-level 
coordination of assessment in states, and specifically, NAEP's roie in 
fetate assessment. The task force will study four broad areas related 
to the revision of NAEP for such comparisons: roles and 
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responsibilities of ED and the states; subject areas to be assessed; 
links to international and other assessments; and sharing of costs. 
3^!J: -^of!? concerns two issues: (1) What should be the roles and 
responsibilities of federal and state education agencies (SEAs) in 
reformulating NAEP to facilitate comparisons of states? and (2 What 
are the problems and solutions for aligning or accounting for 
differences between national assessment objectives and state and local 
curricula? 

The purpose of this paper is to address these two related 
questions as input to the Study Group on National Assessment. In the 
process of doing so, we will attempt to anticipate implications of 
reformulating NAEP for state-bi^-state comparisons for the bidders on 
T^tJ'f^ contract, ED, and ste.te and local education agencies 

i^ich "^^^i^der of the paper is organized into three sections 

1. Postulate that NAEP will come to be viewed as a "national 
achievement test" (for states rather than for individual 
students), and as a result, NAEP objectives will become viewed as 
representing a "national curriculum;" 

2. Recommend that guidelines and procedures for aligning NAEP 
objectives and state and local curriculum and assessment 
ob]ectives should preserve and enhance "curriculun coherence"- 
and ' 

3. Identify instructional and other responses state and local 
instructional staff are likely to make to the use of NAEP in this 
way. This last section also contains further implications and 
considerations for ED's use of NAEP for state-by-state 
comparisons. 



NAEP as a "National Ach'ievement Test" 

NAEP, or any other assessment that may be used for comparinq 
states, is likely to become viewed as a "national achievement test" 
Jo^f ?° "^fP'^effpt to many educators a "national curriculum" 

de facto . It is also likely that states - those that compare both 
livorabiy and unfavorably to other states and the nation -wUl^in 

ntft?.^ r"^^"'^ ""^ ^^'^ curriculum because it is assessed an^ 
used as a basis for comparison and evaluation. 

<c-t ^rivij;'.%^'T^'''^>?^ of evidence to support the notion of a 
i.e:,t driving" school curricula and classroom instruction. For 
example, many state-mandated minimum-competency tests (MCTs) were 
devised to ensure that basic skills curricula be tai>ght and mastered 
one of the purposes of these MCTs h^s been to drive local cur^fcuJ^ ' 
and they have afforded state education agencies (SEAs) and even state 
legislatures unprecedented influence on day-to-day classroom 
activities. Even tests not intended to influence local curriculum 
have done so. -me College Entrance Examination Board's Advanced 
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wf^JS'J**' Examinations ace one such example. AP tests were 
intended to exempt high achieving students from introductorv-level 
^^«:s^"rH^"i" "encourage- high schools wito st^^acadSic 

Sstsr^i™ 

cne rise oe curricula and classroom instructional rpsnnncpc ^,^r. ,Z „ 
the test is mandated far from the classroom e^ l^J^ akd ^L 

t^.^^nr.^fJT^^'^'^^^ °^ ^ address two issues that are 

" F?rst^'S^^" °^ Lff^°""i'if ^= a -national achiev^ent 

^^rHnnifn. ^ """^ ^^^^^ ^^^1 c^ne to define a national 
stati S.?^1oST c^ •'^f be reformulated to assure that its impact on 
achiev^ent° Lrn"?"^? "^11 ^T''^^ curriculun, instructionfand 
aopendSe to exU^^'^^^'iy'-, circumvent NAFlP's becoming an 

TMc. nu^iw ^ose programs/ becond^ in what ways are SEAs and 

^ f "n^M^"^?^^ curriculum and instruction in Response to ^p's 
^pf"°"^^^^^^^^^^"*^ " should becomfclear uSt we 

^WZl ^lrt^t,^lT.l^^^^^^^ -Pa^^-s Should prcote as 



NAEP and Curriculum Coherence 

nnrr,-J!Jl?L°*'^T''''p''^^ ""^^^ "^^^"^ intended to define or describe a 
ma^e ?t SffSufrf; originally deliberately designed to 

ofdistrict" Sg Ls^^to S^ [NAEP] results to state 

/Messick Bea^rS r r 'lo^ ; ^ practices in the schools" 
irSfi p' : ^ P: ^ NAEP Executive Director 

^f^ iAPp'oMEi-*"^ P"*^^ the curriculun business, 

^r^I ^^-^^t^° ^"^^^^ "^'^^'^ the mill" (K. Ashworth, 
^iTttierS et''°"' J'"'^- indeed, NAEP contend area 

publications^ "^^ frequently requested of NAEP 

^. h.I!f ^en^P^ ^^f°'*^^ ^^^^^ °^ the curriculum business appear 
s?a?^by5tatf ^S"^- • 1?" assessments for^^ 

?!tr!7S^ State comparisons may create a new link to teaching and 
learning, as we asserted earlier. The proposed use of NAEP 

acrSrSaL?'af f ^^^r^*^^*^^ comparisons^uld standaSTze curricula 
across, states, at least to some extent. The precise degree of this 
standardization remains to be seen, but the study group^migh? give 
foJi/cSrr??;^^' ''^^'^ potential impact on exiltin^ stite InT 
local curricula in particular, could it lead to misallocation of 
resources? ^aght it undermine whatever degre^ of coherence ?Hpf,-n^ 
below) that now exists in local curricala?''?:r?fcuS^en^e Is 
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closely related to ShuLnan's (1986) research on teacher knowledge of 
subject matter and Tyler's (1984) work on curriculun sequence and 
student learning. Their conclusions suggest that the more coherent a 
curriculum is, the more effective is instruction and learning. 

In our view, coherence extends beyond calls for greater balance 
among subject areas (e.g., Powell, Farrar, & Cohen, 1985) and 
avoidance of "parallel" curricula such as SAT preparation courses 
offered in high schools. While we recognize the significance of these 
two conventional concerns, three other criteria for ensuring 
curriculum coherence also require consideration: organization, 
comprehensiveness, and inter-relatedness. The degree to which any 
curriculum meets these criteria, in large measure, determines the 
degree to which it is likely to facilitate good instruction. 

"Organisation" refers to how the substance of subject matter — 
its major facts, concepts, generalizations, skills, and values — is 
structured. "Comprehensiveness" refers to the scope and coverage of 
all of the most important facts, concepts, and skills. For example, 
objectives from the 1985-86 NAEP science assessment cover not only 
science disciplines such as chemistry and physics, but also processes 
and skills such as the nature and methods of scientific inquiry. 
Finally, "inter-relatedness" refers to the connections forged among 
the parts of a curriculun. A well inter-related American history 
course, for example, would include not only ti\e causes and outcomes of 
the Civil War and Reconstruction, but also the connections between 
these causes and outcomes. (Of course, inter-relatedness can extend 
across subject areas, such as treating pollution in both science and 
social studies courses.) 

We believe that the notion of curriculum coherence, because of 
its importance to instruction and learning, should be a criterion for 
making decisions about reformulating NAEP assessments and objectives 
and for aligning state and local curricula with NAEF. In whatever 
ways sets of NAEP objectives will be recast to meet the demands of 
state assessment and comparison, the coherence of these sets of 
objectives should be preserved and improved. Likewise, in whatever 
ways state and local curriculum and instructional staff adopt NAEP 
objectives or reformulate their own objectives to respond to 
state-by-state comparisons, coherence should be preserved, m this 
case, the coherence of NAEP objectives should be preserved as they are 
adopted into state and local curriculum guides and into instruction; 
and the coherence of state and local objectives should be preserved' 
and enhanced as they are reformulated to align with NAEP objectives 
(We discuss probable SEA and LEA responses to the anticipated 
"national curriculum" below. > 

Until recent proposals for the use of NAEP for comparing states, 
the relationship of NAEP objectives to local curricula has been 
negligible. We predict that debates within NAEP Learning Area 
Committees about the inclusion of particular objectives in an 
assessment will be fired with a new intensity because of the proposed 
state comparisons. Reaching consensus about objectives and exercises 
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is likely to become more difficult as the views of SEA and LEA 
representatives are added. Our admonishment to preserve coherence at 
all levels will only complicate matters further. However, we believe 
that LEA and SEA input should be sought, consensus achieved, and 
coherence preserved so that NAEP assessments and state comparisons can 
encourage instructional improvement. 

The CCSSO, in its own efforts to specify content areas and 
obDectives for assessment and state-by-state comparisons, has 
identified three possible approaches to reaching consensus on what to 
test. ED can consider similar approaches for reformulating NAEP. 
Briefly, these approaches are: 

1. Least common denominator : identify and test whatever 
curriculum objectives are now common across states. 

2. Optimal consensus : identify and test the widest possible set 
of objectives that all states can agree upon. 

3. Common core with local options : identify and test a set of 
objectives all states can agree upon and allow states to 
select independently other objectives to be assessed (Selden. 
1986, pp. 7-8). ' 

In our view, the least common denominator approach would lead to 
a focus on a narrow range of content. Hiis content could be the most 
trivial and unconnected parts of curricula, and those parts already 
mastered by the majority of students. The comnon core/local options 
approach seems a more productive but unfeasible approach. Ebr 
example, states' local options could be so different that they could 
be confusing when comparing states. The optimal consensus approach is 
most likely to produce test objectives that will provide achievement 
data for comparing states and improving achievement. And although 
this approach may make reaching consensus amorg NAEP staff and 
consultants and SEA and LEA representatives difficult, it is the one 
most likely to preserve coherence of NAEP and local objectives. 

Some procedure for reaching consensus on what to test will have 
to be developed or chosen. Whatever that procedure may be, it should 
incorporate current NAEP procedures for generating, reviewing, and 
selecting NAEP objectives and exercises. These procedures include 
various steps for reviewing and revising objectives and exercises and 
Sor input from subject matter and measuranent experts around the 
country and within the NAEP staff, and by staff of the Office of 
Educational Research and Improvement (Educational Testing Service 
1984). The procedures result in sets of coherent objectives Tb 
illustrate, NAEP objectives booklets that we have examined — from the 
1983-84 writing assessment, for example — meet our criteria for 
curriculum coherence. Objectives for this assessment are organized by 
purpose and audience (personal wri r-^- that students may use to learn 
about acadenuc subjects and themselves, and public writing that is 
intended to inform, persuade, or entertain) and inter-relate these 
purposes by assessing students' control of the writing process 
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language forms, and conventions (NAEP, 1982). 

In addition to continuing and incorporating these procedures, we 
also recommend strong representation of staff fran SEAs and LEAs. 
Such representation is essential if a good match among NAEP 
objectives, state curriculum guidelines, and local curriculum and 
instruction is to be fostered. This juncture in the evolution of NAEP 
and state assessment may be a once-an-era opportunity to encourage and 
aid student achievement from the federal level. Input from SEAs and 
LEAs can only help take advantage of the opportunity and preserve 
curriculum coherence. 

Some discussion is appropriate at this point of NAEP score scales 
and their relationship to NAEP objectives. ETS began the practice of 
reporting student performance on NAEP assessments using score scales 
such as the Levels of (Reading) Proficiency (ETS, 1986) . NAEP reports 
describe in a general way what students at each proficiency level are 
able to do. For example, students who read at a "Basic" level "can 
locate and identify facts [and] . ..understand specific or sequentially 
related information" (ETS, 1986, Figure 2.3). However, these levels 
are not clearly tied to NAEP objectives or to any implementable 
curriculum. Reporting performance on such scales facilitates 
criterion-referenced-like interpretations of student performance. 
But it would not capitalize on the opportunity to go beyond only 
comparing states on valued instructional outcomes to encouraging 
their performance on these outcomes. To fully benefit from the use 
of such scales in reading and other assessments, reports on levels of 
proficiency should be accompanied by the NAEP objectives related to 
each scale so that they can be implemented into state and local 
curricula. We do not mean to build an argument for ranaking NAEP 
assessments into domain or criterion- referenced instruments from which 
strict interpretations can be made about the portion of a domain of 
skills and knowledge students have mastered. We are encouraging, 
however, that new procedures for reporting NAEP results include 
publicizing the link among proficiency levels, NAEP exercises (test 
items) , and the coherent set of objectives used to guide exercise 
development . 

To summarize, although NAEP claims not to be in the curricultm 
business, it may enter if NAEP assesanents are used for state 
comparisons. What subject matter will be assessed on iJiese "national 
achievement tests" is critical because it may become a "national 
curriculum." Because of its importance for effective instruction and 
learning, this curriculum must be coherent. Strot^ SEA and LEA 
representation and input to the content of NAEP assessments can 
preserve and even promote coherence. And clear descriptions of the 
links between NAEP results and objectives assessed by NAEP could 
improve instruction and student learning. 
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Likely Instructional Responses 
by LEAS and SEAs 
and Implications for ED 



progr?n^^m J^?^ ^"^'^ f instructional 

o^r T^dTiiHn^ comparison with the entire country and 

^Id iSve far!refchinn ^ff Any state's comparative ranking 

couia nave tar-reaching effects on education funding levels- miaration 
of businesses, industry, and families in and out of the st^te- a^d 
i?e efo4 °' flj-being and satisfaction wl^hlheir 2ho^S. 

3ill Sjan Ji.Wn ^5^^?^^ to conclude that state and local staff 
Will align their curricula with what is tested, what follows is a 
list of responses state and local educators are likely to m^e to 

can^'b^'tSiaht ofT"!^ comparisons. The Lin'artrdepicted 

can De thoiight of as what we might consider doina as «?ta^e or lo^ai 
school superintendents, in the face of bei^g ^r^ to other sS?e3 
and the nation. Members of the task force Say discoverfurther 
considerations in these likely responses for Lp's use foTS^paring 



Policy Responses 



State and local educational policy-makers may choose to align 
^SrJicS?^ ""i^^h2^ instruction programs with this new "national 
either aSot -nay choose not to resjwnd. Hiat is, they may 

eicner adapt, adopt, or ignore NAEP objectives. 

Adapt NA5P objectives to state or local curricula , or vice-versa 
. Some superintendents of instFUEti^H iiiiTTi nd that the TF cur^nt 
programs overlap with sets of NAEP objectives, and that ^e additions 

statf^nH ?oLf^o Objectives is representative of 

taalt f .'^^r^^!;'^^ ""^y ^ary widely from state to state and 

local district to local district, m turn, the degree of 

rSuirS'r^''^"^- ^ ^^^^^y influence tow muc?adap?f tion is 
ISoi !f ^'^ ^^^^^ and local districts. Further: ?Se 

adaptations that are made could diminish the coherence of N^EP 
ob^ectiyes while preserving the coherence and local relevance of 
r^^.^r!'^^ °' vice-versa. As a result of adapta??^n, SA^^ Sbjectiies 
could become curriculum appendages, with litUe apparent c^nneS on to 
other parts of the curriculum. Such a parallel curriculum^would no^ 
m^ri^2K'"'''°" ^^^i-^ example of ^cZlfry 

Adopt NAEP objectives . other superintendents may see the 
ev^o"tL?r'"'" opportunity - or even a Sdate -to 

revamp thexr curriculum. In fact, it is possible that many states and 
v,?n« '^'^^ ^° ^° ^ ^" °"^^er to align themselves with 

NAEP objectives -mis need may be especially true for assessments of 
subject areas that do not receive wide-spread curriculum attention 
(such as citizenship and art) or in which instructional approaches 
have changed in recent years but have not been implemented in all 
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states and schools (for example, the writing orocese annr^^^h 
especially In states that do'noi ^ve"di reel imV^ ^s^Ill^^ts, . 

Ignore NAEP and state comparisons . Of course «;ta^^»e: an^/«K 
S^? on^a'lS^ ^ir?„^raiFisons to^othlr ^ iT^^ 
;,3oni f f,KSi^''^^"^*^^^*^• ^^^^^ choose not to adapt to or 
It ll.l/J^'^ objectives In addition, locals may choose^o adapt to 

objectives in lesser or greater degrees than is their 
state department of education's policy. 

Whatever decisions are made about aligning state and local 
curricula with NAEP objectives, such decisions should be guided bv 
considerations for their impact on curriculum coherence and 

^Ihf^M^ "tL^^i?''^^''^"^^^- '"ig'^t be wise to 

publicize with which NAEP assessments and objectives their state and 

i;2i^°;i^'f^"^^ ^"^^ ^^^^ constituents not IT 

expect performance on assessments in tliese areas. 

into ffcal Svit^Ss"?^ .^^f i-^Pl^^^ntation of innovations 

^!?kes^-SiS^^ 

in««ded by thif a^apCiSo^ pJo^eS^^ ^iSti^^^ ""^'y *^ 
procedures do not allow staK= I f^fther, current NAEP sampling 

districts will partici2L?^?n a^rH ■'JP^'^ ^^^^ ^ 

results could oroduce » „f.i J Particular assessment. Assessment 

positive or nela'uvfdfr^UoT 11,?'%""^%°' ^ P^rticul« sS^fin a 
SSr%1e%n^ed-»«?"^^^^^^^^^^^ 

but M.cH cHose L i,no=?^ ^11«er[n''&i^-,----t 
that *"-n«rona^"t;^cu^™^Tb^ft?^ observations? First 

xrszrta^i^j pic£t^^^^^^^^^^^ 

nstruction and studen? acSent Pinaflv'' f^^^'^ °« local 
integrate NAEP objectives uJhh^ ""f"!"' local efforts to 
supported through tecinilafSsf S <=""'<="1= Probably should be 
universities and oSer staiel^id^""'/.'^."? ^ """^ 
coherence is preserSid !^d :Ln1ed ^ t^at curriculun 



Practical Responses 



^.P- ^5 t^^^^^ ^"'^ ^° ^h°°se to rf.^oond to the curricnl.n, 

actWi^ier:"^' objectives, they could resjx^nd w?th''a%'Jeasrfo^r 
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1. Conduct curriculum- test match studies to identify curriculum 
areas assessed by NAEP not included or emphasized in local 
curricula. What could result is shifts from what is currently 
being taught to what "should be" taught in order to prepare 
students for NAEP assessments. The shifts could improve or 
diminish curriculum coherence at the local level, <a-; we discuss 
above. 



practice tests — generated from NAEP 
objectives and similar to NAEP exercises — to iden^ify areas of 
strength and weakness. This sort of response to NAEP was not 
originally intended, but represents an opportunity for NAEP to 
encourage achievement, not just describe it. However, such a 
response could be in direct competition with other local 
assessment activities, either for assessment time or for 
curriculum emphasis. 

3. Review textbooks and curriculum materials for adoption in liqht 
t.??®^'^ coverage of NAEP objectives, this in turn could affect 
publishers' revision, development, and marketing activities 
similar to the effects of the basic skills movement of the late 
1970s. 



4. Conduct workshops: SEAs could provide training and assistance to 
LEA staff, who in turn could aid school staff in integrating, 
adapting, and implementing NAEP objectives. 



Conclusion 



Some of the responses and implications we suggest may seem 
far-fetched at this time. For example, local curriculum staff and 
teachers may seem well-insulated from the rumblings an SEA may make as 
a result of being compared to nearby states. The notion of NAEP as a 
"national achievement test" representing a "national curriculun" may 
seem mildly implausible at best. However, we should keep in mind that 
as recently as the early 1980s state legislators and departments of 
education probably had no idea of the effects they would have on 
classroom activities by tying high school diplomas to performance on 
minimum-competency tests. It takes only a bit of imagination, we 
believe, to envision the similar impact that using NAEP for 
state-by-state comparisons can have on state and local curricula and 
achievement. 
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